Multi-objective discounted dynamic programming The Neighbour Search approach to construct Pareto sets of multi-objective Markov Decision Processes

The Neighbour Search (NS) algorithm, is an iterative method for constructing Pareto sets of multi-dimensional polytopes. A NS iteration consists in two steps: Edges Exploration and Neighbour Detection. Edges Exploration takes a Pareto vertex and determines all Pareto edges connecting such a Pareto vertex to its neighbours. Each neighbour is again a Pareto vertex that is obtained by Neighbour Detection. The procedure continues until all Pareto vertices are explored. The purpose of this paper is to describe in detail the application of NS to Markov Decision Processes (MDPs) with N discounted objectives. Novel numeric techniques are herein developed to effectively adapt Edges Exploration and Neighbour Detection to the MDPs characteristics. Edges Exploration consists of solving a problem of redundancy removal for systems of linear inequalities in N dimensions; the number of inequalities is equivalent to the size of the MDP. Neighbour Detection is performed either by Direct Neighbour Search (DNS) or by Cross Neighbour Search (CNS). The former requires the Bellman equation to be solved, even though with a reduced action set. The latter does not require the Bellman equation to be solved, and is computationally linear in the size of the MDP, and thus more efficient than DNS. However, CNS requires conditions that are not always fulfilled, whereas DNS is always applicable. Experimental results suggest that conditions for CNS to be applicable are actually satisfied for the most of NS iterations. In Gianluca Dorini Department of Environmental Engineering, Technical University of Denmark, Miljvej 113, DK-2800 Kongens Lyngby, Denmark E-mail: [email protected] Dragan Savić Centre for Water Systems, School of Engineering, Computing and Mathematics, University of Exeter, Harrison Building, North Park Road,Exeter EX4 4QF UK, E-mail: [email protected]